Systems and methods for integrating 3D diagnostic data

ABSTRACT

Systems and methods are described for integrating anatomical information from a plurality of sources of information. The system receives two or more three dimensional (3D) anatomical maps sharing a common plane specified by three or more marker points common to the two or more maps; and aligns the two or more 3D anatomical maps using the marker points.

This application is related to U.S. patent application Ser. No.09/560,424, entitled “Systems and Methods for Generating An Appliancewith Tie Points.” The present invention relates to three-dimensional(3D) representations of objects such as anatomical structures.

BACKGROUND

Many applications, including medical and dental applications, rely ondata visualization to provide a holistic display of information and tomaximize the amount of information that can be conveyed at once. In datavisualization, a graphical mapping from an information space to adisplay space is performed. The process allows a user to visuallyexplore the 3D representations or models of objects. The 3D modelsbecome much more useful when they are integrated, that is, when a numberof related models are merged and overlaid to provide spatial context,among others.

For example, knowledge of the spatial relationships of the jaws, teethand cranium is needed in various dental applications. Such relationshipsinclude the relative positions of hard structures (teeth and bones) andoverlying soft tissues and skin. The customary types of physical recordobtained by dental clinicians include photographs of the face (bothextra-oral and intra-oral), X-ray images of the skull taken fromdifferent projections, and plaster study casts of the teeth themselves.Conventionally, most diagnoses are currently made using 2D photographs,2D X-ray films and 3D plaster study casts.

SUMMARY

A system integrates anatomical information from a plurality of sourcesof information. The system receives two or more three dimensional (3D)anatomical maps sharing a common plane specified by three or more markerpoints common to the two or more maps; and aligns the two or more 3Danatomical maps using the marker points.

Implementations of the system may include one or more of the following.The anatomical information can be stereo craniofacial data. One of theanatomical map is an X-ray map. The X-ray map is generated usingcorrelated points on X-ray pairs and using y-parallax measurements. TheX-ray information is stereo. The system can calibrate one or more X-raysources. The calibration determines a principal distance from an X-raysource to a film plane. The system can also characterize internaldimensions of the one or more X-ray sources by locating an X-ray filmrelative to an X-ray source. One of the anatomical maps can be a 3Dimage map. Another anatomical map can be a dental map. Each marker canbe a tie point. The system can also use discrete anatomical landmarkinformation. The system can display the aligned maps as an integrated 3Danatomical model.

In another aspect, a method visualizes anatomical information from aplurality of sources by receiving X-ray information having X-ray markerinformation; receiving a three-dimensional anatomical information havinganatomical marker information; receiving a three-dimensional teeth modelhaving teeth marker information; aligning the X-ray information, 3Danatomical information, and the 3D teeth model using the markerinformation; and displaying the aligned X-ray information, 3D anatomicalinformation, and the 3D teeth model.

In another aspect, a system includes an X-ray camera receiving X-rayinformation with X-ray marker information; a three-dimensional digitalcamera receiving three-dimensional anatomical information withanatomical marker information; a dental scanner to generate athree-dimensional teeth model with teeth marker information; a computerto align the X-ray information, 3D anatomical information, and the 3Dteeth model using the marker information.

Advantages of the invention may include one or more of the following.The system improves the capability of orthodontic practitioners todevelop diagnoses orthodontic treatment plans.

Through the use of markers called tie points, the system allows mergingof multiple data sources such as X-ray images, study cast images ormodels, and facial images into a single unified coordinate system.Because the coordinate frame of reference is unified, two or more datasources that are not directly tied together can in fact be compared. Forexample, because the study casts are referred to the X-ray anatomicframework, and the facial imagery is tied to the same X-ray anatomicframework, therefore the study casts and facial imagery can be comparedto each other, even though they are not directly tied to each other.Overall, the system also provides a uniform accurate coordinate systemin three dimensions, meaning that accurate unbiased measurements can beobtained, regardless of subject orientation.

The system permits each type of anatomical structure to be imaged usingan appropriate imaging modality and perspective best suited for locatingthat structure, after which all the optimal locations can be merged intoa common 3D map by using tie points. The system is robust inconstruction, simple to operate and requires a minimum departure fromthe technician's usual procedures. The user only needs to identify andlocate landmarks, after which all measurements, computations, andintegrations of data from different primary sources are madesemi-automatically or automatically by computers and computer-aidedtechnicians.

The ability to tie the data sources together opens up new diagnostic andtreatment planning opportunities. The system supports a dynamic analysisof the functioning of the jaws, teeth, and musculature.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing a system for performing integratedthree-dimensional craniofacial analysis from a combination of datasources

FIG. 2 shows the sequence of processes by which integrated 3D graphicaland numerical data are generated for storage, recovery and utilizationby the database shown in FIG. 1.

FIGS. 3A, 3B and 3C illustrate top view, front view and perspective viewof an embodiment of a stereo X-ray machine.

FIGS. 4A-4F illustrate an exemplary calibration array.

FIGS. 5A-5B shows a cassette carrier or holder.

FIG. 6 shows two views of a tie-point fitting which a clinician can bond(attach) directly to a tooth.

FIG. 7 shows a type of tie-point fitting for use on the face of apatient.

FIGS. 8A and 8B illustrate radiographic markers that are embedded in anappliance.

FIG. 9 illustrates an exemplary mounting of a number of tie points on asubject's face.

FIG. 10 is a diagram showing a data management system supporting theintegration of three-dimensional craniofacial analysis from acombination of data sources.

FIG. 11 is a simplified block diagram of a data processing system usedto generate an appliance.

DESCRIPTION

FIG. 1 shows a system that enables digitally encoded graphicrepresentations the teeth and the facial surface to be merged withaccuracy and precision, making possible three dimensional diagnosis,treatment planning, and evaluation of results in orthodontics and othertypes of dental and craniofacial treatment.

Referring now to FIG. 1, a set of radiopaque and photographicallyvisible spherical tie points 12 are temporarily mounted on the surfacesof the face and teeth of a patient 10. With these tie points 12 inposition, two or more skull X-ray images are generated from differentperspectives using an X-ray camera 20. The camera 20 can be a stereocamera with two X-ray emitters, or can be a calibrated single emittersystem. The X-ray images can utilize either coplanar or biplanargeometry. Each tie-point 12 is unambiguously visible on two or moreradiographic projections. The spatial relationships between the X-rayemitter(s) and the X-ray film or other image-capture medium are alsoknown with accuracy and precision. The X-ray camera 20 is calibratedwith a specialized calibration apparatus and associated software. TheX-ray camera 20 uses a cassette-holder installed at the opposite end ofthe X-ray camera from the X-ray tube.

The skull X-ray images can be of either the bi-planar or co-planar type.If they are of the coplanar type, three-dimensional locations ofstructures not lying in the sagittal plane can be obtained. Thus, 3Dwire frame representations of skull and jaw anatomy can be constructedin a manner similar to the 2D tracings of traditional orthodonticcephalometrics.

A 3D image-capture system 30 captures digital 3D images of both 1) thetooth surfaces and their associated tie points and 2) the facial surfaceand its associated tie points. Particularly, three or more measurabletie points are available in common for each pair of 3D images to beintegrated or merged.

Additionally, an appliance 50 is mounted on the patient's teeth. In oneof these implementations, the tooth tie points are temporarily cementedto the teeth prior to taking the impression for the appliance 50. In theother implementation, the impression is taken first and the tie pointsare incorporated into the appliance 50 during its fabrication. Anintegration system 70 receives data from the X-ray camera 20, the 3Dcamera 30 and a digital model of the appliance 50 and integrates thedata to provide a holistic view of the patient 10 for treatment. Theintegration system 70 is shown in more detail in FIG. 10.

An exemplary use of the system of FIG. 1 is discussed next. First, thephysical patient records are acquired. During this process, a treatingprofessional such as an orthodontist or otherwise secures four to sixbonds radiopaque metal spheres to upper canines and molars and fourradiopaque metal spheres to lower canines and molars. The metal markersare called “tie points” such as are shown in FIGS. 6-8B. They will laterbe used for linking data from images of different types into a commonthree-dimensional coordinate system. High quality upper and lowerimpressions are then taken, suitable for destructive scanning, forexample. Additional tie points, like the one in FIG. 7, are attached tothe facial surface prior to taking X-rays and facial photographs in aconfiguration similar to that shown in FIG. 9. Lateral and frontalstereo X-ray images and a single panoramic X-ray image are taken withthe tie points in place, using a suitable digital or analog X-raysystem. Frontal and lateral 3D facial photographs are taken using a 3Dphotographic system. Fully digital versions of all the above records arestored in the integration system 70.

In order to integrate the data from all sources the positions of the tiepoints on each physical record must be located. On study casts such asthose made in accordance with U.S. Pat. No. 5,975,893 issued to AlignTechnology, Inc., the 3D data are generated by the Alignsoftware/hardware system. On the facial photographs, the procedure canbe done by an operator on screen, also directly in three dimensionsusing a suitable 3D camera. The reconstruction of X-ray information ontie point locations must be done individually on each image as anon-screen operation. However, it can be done by a minimally trainedtechnician or assistant and does not require any orthodontist's time.Automated digital pattern recognition methods may also be used to locatetie points on images without human intervention.

For the three space coordinates of an anatomical landmark on an X-raycephalogram to be precisely known, the landmark must be independentlylocated on both images of an X-ray stereo pair. This step requiresinvolvement of either an orthodontist or a trained technician or dentalassistant. Since the tie points are in the same position as the faceduring the taking of both the lateral X-ray images and the frontal X-rayimages, all anatomical landmarks can be methodically transferred anddisplayed in the proper anatomic relationship to each other. Theexistence of the tie points allows landmark information from lateralimages to be precisely related to information from frontal imagesdespite changes in position of the head between stereo pairs. Inaddition, when information from a panoramic X-ray image is combined withinformation from the study cast map, the positions of tooth roots willbe able to be located with only slightly reduced accuracy and precision.

At this point, the treating professional has available a fullyexplorable multi-layered representation of the skull in threedimensions. Measurements from the teeth to each other and to thesurfaces of the face can be performed using a mouse-driven cursor. Ifthe treating professional desires conventional two dimensionalcephalometric information, he or she can obtain it by locating landmarksand surfaces solely on the centered lateral and centered frontal imagesfrom the emitter 310 without the emitter 312. In order to obtain fullskeletal information in three dimensions, the treating professional or atrained assistant will be required to locate the same landmark on eachof two stereo X-ray images using the emitters 310-312.

In one implementation, to assist in relating the data from the datasources, radiopaque and photographable metal spheres are attached to thesurface of the face while similar spheres are mounted on temporaryremovable plastic templates which are formed on the study casts and arereadily transferable between the casts and the mouth. All thesereference markers are mounted on the face or in the mouth at the timethe appropriate stereo X-ray pairs are exposed. Following processing ofthe images, the tie points coordinates are located and mapped in afashion identical to that employed for any anatomical landmark ofinterest. Landmark locations on the final maps are usually expressed interms of an anatomical coordinate system based on the lateral skullX-ray film map. After the various individual coordinate maps for theX-ray films, study casts, and facial photography have been generated,they are integrated with each other through a series of mathematicaltransformations. After these transformations, the resulting integratedmap can be visualized, manipulated and measured on a computer monitor.

The information stored in the integration system 70 is merged to createa holistic view of the patient data from a plurality of data sources.FIG. 2 shows a process 199 which uses information from the raw dataacquisition module 102 and database 104 of FIG. 1. First, an appliancecontaining radiopaque tie points or markers is created. The tie pointsare identifiable markers that are visible in multiple types of images,or across multiple time points within a single type of image. Thelocations of the markers are specified in advance. When digital camera3D data is merged with stereo X-ray data, common points in both sets ofimages are needed. In this case, small radiopaque spheres aretemporarily placed on the face of the patient. They show up as small“targets” in the digital facial images. Because they are radiopaque,they also show up in the X-rays. Then 3D coordinates are computed forthese common tie points from both the stereo X-rays and from the digitalfacial photographs. Once the coordinates are known from both sources, amathematical transformation can be applied to rotate and translate thefacial data into the same frame of reference as the X-ray data.

In the process 199, the radiopaque markers are then positioned on thepatient's teeth at their designated locations using the aligner (step200). At the same time, the patient is also fitted with a second seriesof tie points that consists of radiopaque markers on the face. Next,stereo X-ray images are acquired using a calibrated X-ray machine (step202). Lateral and frontal images are digitized, and three dimensionalcoordinates are computed for the tie points on the teeth, the tie pointson the face, and any desired anatomical landmarks visible on any two ormore X-ray images (step 208).

From step 202, 3D facial data are acquired using a 3D camera (step 204).Next, 3D coordinates for facial Tie Points from facial digital image arecomputed (step 210). From step 204, one or more appliances are created(step 206), and 3D coordinates for tie points on teeth computed from theappliances (step 212). The 3D coordinates are computed from the studycasts for the tie points on the teeth. At this point, 3D coordinates areavailable for: a) anatomical features from X-rays; b) tie points on theteeth from X-rays; c) tie points on the face from X-rays and facialimages; e) tie points on the study cast and therefore the model of thedentition.

During this process, the correspondences between the markers on theteeth and their images on the X-ray images are identified. Also, thecorrespondences between the markers on the face and the images of themarkers on the X-ray images are identified. Using the two pairs ofcorrespondences, the three data sets are mathematically registered in 3Dspace as a computer operation so that the corresponding tie pointscoincide. The integrated map can now be viewed in any desiredorientation.

Once aligned, the system can display all three data sets together withthe upper and lower tooth images. The face image is correctly positionedwith respect to the X-ray image. The stereo X-ray and the resultinganatomical coordinates act as a framework or scaffold upon which theother data sources (study cast and facial images) are hung. In this way,the study cast data and the facial image data can be visually andanalytically compared to each other, even though there is no direct dataconnection between the two.

3D coordinates of tie points on the patient's teeth and face (from X-rayImages) are transformed to an anatomical framework (step 214). Further,3D Coordinates of Tie Points on the study casts are transformed to theteeth tie points from the stereo X-rays (step 216).

From step 216, 3D coordinates of tie points from the patient's face(from facial digital images) are transformed to the facial tie points(now in the anatomical frame of reference) from stereo X-rays (step220).

In sum, two appliances representative of the two jaws are fabricatedupon the study casts of the upper and lower dental arches in such amanner that the teeth can occlude freely and without interference withthe appliances in place. Each appliance has fastened to its surfacethree or more radiopaque metal markers called tie points. The sametechnique is used on the facial 3D photos to define a known plane in theface by the expedient of placing three or more radiopaque andphotographable markers (tie points) on its exterior surface. The stereoX-rays of the skull are made with the appliances in place in the mouthand with the facial tie points in position. Because of the radiopaqueproperty of the tie points, they will also be unambiguously identifiablein the stereo X-rays of the skull.

In one operating approach, a clinician bonds tie points to teeth using afitting such as that of FIG. 6. Clinician then takes one or moreimpressions of the patient during a records session. The impressions areused after completion of the records session to generate an applianceand coordinates of the teeth and tie points are generated during themanufacturing of the appliance. The clinician then takes stereo X-rayimages and 3D images of the patient. Alternatively, in lieu of thestereo X-ray images, conventional bi-planar (frontal and lateral) X-rayimages generated from only one emitter can be utilized, provided thatthe X-ray system is calibrated and that the dental and facial tie pointsare unambiguously identifiable on the lateral and frontal X-ray images.In such an application, however, 3D location of anatomical structuresother than those lying in the sagittal plane cannot be effected.

In a second approach, the clinician takes appropriate impressions. Anappliance is then created with tie points embedded at known locationswith respect to the teeth. The appliance containing the tie points isprovided to the clinician who positions it in the patient's mouth. X-rayand 3D facial images are then captured, as discussed above.

FIGS. 3A, 3B and 3C illustrate top view, side view and perspective viewof an embodiment of a stereo X-ray machine. The stereo X-ray machine hastwo emitters 310-312 that are mounted on a horizontal bar. In thisembodiment, the emitter 312 is positioned 18 inches from the emitter310. The emitter 310 is positioned 60 inches directly from a patient316. The patient 316 is positioned in a cephalostat (not shown)approximately six inches in front of a cassette carrier 318 (FIGS.5A-5B) which defines the datum plane. The emitters 310-312 and thecassette carrier are vertically supported by a support stand 314. Theemitter 310, the cephalostat and the cassette carrier are offset fromthe stand 314 by about twelve inches. The stand 314 allows the X-rayequipment to be adjusted to the height of the patient 316.

This embodiment includes a rigid rectangular structure with twin HRT1General Electric X-ray emitters 310-312 mounted at one end. At theopposite end of the stereo X-ray imaging system, beyond the patient'shead is a machined cassette holder defining the datum plane 318. Theemitters 310-312 and the carrier 318 taken together constitute the“cameras” of the stereo system. The relationship between the cassetteholder 318 and both X-ray emitters 310-312 is known through previous useof a calibration device. The calibration process consists of imaging acalibration cage with three planes of precisely known radiopaquetargets. Using a simultaneous least squares adjustment (“bundleadjustment”), the 3D locations of the x-ray emitters are determined.

The mathematical calculation of the 3D positions of the tie points andanatomical structures on the stereo x-ray images depends upon havingaccurate and precise information on the physical relationships betweenthe focal spots of the two entities (310 and 312) and the surface of thex-ray film or its digital equivalent located in the cassette carrier at318. The information needed includes: (1) the distance between the twofocal spots 310 and 312, measured parallel to the surface of the film inthe cassette carrier; (2) the perpendicular distance between each focalspot and the plane of the film surface, and (3) the precise location ofthe cassette within the cassette carrier for each exposure. Thislocation differs slightly for different projections as may be seen byexamining the x-ray intersections 316 and 318 in FIG. 3a. To obtain thisdata, a calibrated array, the cassette carrier previous mentioned, andan “auxiliary calibration checking frame.”

FIGS. 4A-4D illustrate an exemplary calibration array 500. An imageablestructure of known dimensions is placed in a location from which it canbe X-rayed. The structure consists of a radiolucent framework upon whichare mounted a number of radiopaque points whose three space locationswith respect to each other are known with accuracy and precision. In thesimplified illustrated case there are four perpendicular radiolucentplastic rods 502, 504, 506 and 508 at the top and bottom of which theradiopaque points are mounted perpendicular to each other. When theX-ray image is exposed, the shadow points 503, 505, 507 and 509 of thetop of the rods 502, 504, 506 and 508 will be cast upon the filmsurface, radially displaced with respect to the original three spaceposition of the rod top point. A line drawn in three space which passesthrough both any rod top point and the image of that point on the x-rayfilm will also pass back along the path of the ray which erected theimage through the focal spot from which the ray originated. Thisprinciple is shown in photogrammetry as the principle of resection.Since the same principle applies to all four rods, a series of linespassing through an x-ray focal spot 510 (corresponding to point 310 or312 of FIG. 3A) may be generated, the intersection of any two of whichwould identify the focal spot uniquely if the measurements were withouterror. Since there is always error, redundancy is added in thisimplementation by the use of four rods rather than two.

By repeating exposures for both x-ray sources, (the equivalents of 310and 312), the distance from each x-ray source to the film plane can becomputed. Next, a point 520 on the film plane at which the perpendicularray strikes the point is identified. This point is called the “principlepoint” and the distance between the principle points of both x-rayimages in the precise measurement of the distance between the focalspots of the two x-ray sources measured parallel to the film surface.

To locate the principle point for each x-ray source, a line is drawn inthe plane of the film passing through the images of each rod top pointand its, rod own bottom point. Information from two rods can be used inthe absence of measurement error and the redundant information arisingfrom the use o four rods strengthens the calculations in the presence oferror. At this point, the distance from each x-ray source to the filmplane, the point of contacts on the film plane of the principle ray fromeach source, and the distance between the two x-ray sources are known.

In certain cases, it is impractical to use the system of fourperpendicular rods surrounding the patient's head because of physicalclashes with the cephalostat that is required for the control of headorientation in object space. For this reason, another exemplarycalibration device 518 is shown in FIGS. 4E and 4F. The secondembodiment consists of 3 plastic planes 520-524 assembled in parallellayers, successively separated from each other by intervals of a knownvalue, in one embodiment 9.5 inches. This device is positioned betweenthe x-ray sources and the film plane, as near as possible to it. In thisembodiment, the plane of the device nearest the film plane has nineradiopaque markers in carefully measured locations; the other two planeshave four markers each. While the principle of measurement is stillresection, the redundance of this calibration device avoids the need toorient it with great precision with respect to the film plane. Theapproximate precision in measuring the principal distance with thisapparatus is in the one to three millimeter range. The distance betweenthe x-ray emitters can be measured with one millimeter.

The next task is to obtain precise information about the preciseorientation of the film cassette with respect to x-ray emitters. Forthis purpose, a cassette holder 318 (FIGS. 5A-5B) is rigidly attached toX-ray emitters or sources 310 and 312. The holder 318 has on its surfacea set of precisely positioned radiopaque fiducial points (324 beingtypical) whose precise coordinates are known through directmeasurements. These fiducials, in sets of at least four are imaged onthe film surface during exposure, thus identifying the precise positionof each film with respect to its x-ray source.

Since the control array cannot be maintained in position all the time, aslightly lower order auxiliary calibration checking frame is supplied.This device (not shown) consists of a smaller number of radiopaquespheres mounted on a plastic cone or box in known positions which can beperiodically fastened to a cassette positioned within the cassetteholder. While it is not sufficiently redundant for use as a calibrationarray, it is powerful enough when used periodically to check therobustness of calibration achieved using the full calibration array.

The sequence of steps in the operation of the stereoscopic X-ray systemis as follows:

1. The subject's head is positioned within the object space, secured ina head holder or cephalostat to minimize movement between exposures.

2. A cassette containing an unexposed X-ray imaging medium (film ordigital medium) is positioned in the cassette carrier 318.

3. The X-ray emitter located at a first station is fired, exposing thefilm or a digital image capture medium.

4. The first cassette is removed and replaced with a new cassettecontaining a new unexposed film or digital image capture screen medium.

5. The X-ray emitter located at a second station is fired exposing thefilm.

After producing the stereopair of lateral skull films, the patient andthe cephalostat are next rotated 90 degrees such that the patient facesthe cassette carrier 318. Two more films are exposed from the first andsecond stations. These constitute a frontal X-ray stereopair. The filmspairs are processed and examined by a monocular analytic method in whicha person examines the two images sequentially and marks the locations ofthe cassette holder fiducial points, the tie points, and one or moresets of anatomical landmarks. The location of these structures on theX-ray images can be done manually or by using suitable computerprograms.

Software is then used to examine the tracings for each film as a unit ofinformation. If duplicate tracings have been made, the software rotates,translates and rescales the multiple tracings for each X-ray image to aleast squares best fit upon the directly measured known X and Ycoordinates of the fiducial points on the datum plane. The set ofcoordinate values for all landmarks of each film is stored as a block.The coordinates of the principal point of each film and of the fiducialpoints on the datum plane are added to the set of coordinate values forthe X-ray image.

The coordinate files for the two films of each stereopair are now usedto produce a three dimensional map in the following manner. The file forthe film from second camera station 312 (the offset film) is rotated andtranslated such that its four registration points are best fit upon theregistration points of the film from the first camera station 310 (thecentered film). The X parallax of each point in the system may nowreadily be computed as the ΔX between any landmark on the “centered”film and its conjugate on the “offset” film, and the altitude of thelandmark is computed automatically using standard photogrammetricequations. In addition, the Y parallax for each landmark andregistration point is computed as measure of the degree to which thesame physical structure was actually identified on both films of thestereopair. More information on the transformations is described inElements of photogrammetry, Paul R Wolf, McGraw-Hill (1974). SystematicY parallax deviation for anatomical landmarks which are significantlygreater than those for the fiducials are indicators of patient movementbetween the exposure of the two images of the stereopair.

In one embodiment, the raw data acquisition module 102 receives 3Dfacial data from a 3D camera such as the Venus3D camera available from3D Metrics of Petaluma, California. The 3D Metrics camera uses a 3DFlash light projector and a conventional digital camera. The 3D Flashlight projector projects a color-coded white light pattern onto anobject, the digital camera takes an image of the object, then the datais transmitted to a computer which performs across-talk-free-color-decoding operation. The 2D data is then convertedinto a 3D image. In this manner, the 3D camera or imaging system enablesa single regular digital camera to take a 3D image with only one shotand does not require any additional mechanical scanning, laser lightsource, or additional shots. The 3D data generated by the 3D camera isthen scanned for the location of tie points. In one embodiment, the tiepoints are manually identified by a person. Other embodiments wouldprovide automatic detection of the tie points.

Referring now to FIGS. 5A-5B, a cassette carrier 318 is shown. Thecassette carrier has a plurality of tabs or fiducial markers 324 and aplurality of stop points 322. An X-ray cassette is inserted into thecassette carrier and slid into position behind the tabs 324. The tabs324 are radiolucent and carry radiopaque fiducial control points similarin size to the tie points. However, they differ from tie points in thattheir exact spatial location with respect to the X-ray tubes 310 and 312are previously shown through prior calibration. The stop points 322 areadjustable to limit the lateral traversal of the cassette at variousdesired positions.

FIG. 6 shows two views of a tie-point fitting used with for use with adirect bonding onto the teeth of a patient. The tie point fitting has aplastic carrier 330. Mounted on top of the plastic carrier 330 is aradio opaque tie point 332. As shown in the side view and the edge view,the plastic carrier 330 has serrations at a bottom end to facilitatemounting of the tie points 332 onto a tooth of a patient using asuitable glue or bonding system.

FIG. 7 shows a tie point fitting for use on the face of a patient. A tiepoint 342 is positioned on top of a paper or adhesive tape carrier 340.The carrier 340 has an adhesive backing that attaches the tie point tothe face of the patient.

Next, appliances with radiographic markers will be discussed. In oneimplementation, radiographic markers are embedded in an appliance 300,shown in FIGS. 8A and 8B. The appliance 300 is a polymeric shell havinga cavity shaped to receive teeth. The polymeric shell typically fitsover all teeth present in the upper or lower jaw. The polymericappliance 300 is preferably formed from a thin sheet of a suitableelastomeric polymeric, such as Tru-Tain 0.03 in. thermal forming dentalmaterial, marketed. by Tru-Tain Plastics, Rochester, Minn. 55902. Theappliance 300 has four radiopaque metal pins 302, 304, 306 and 308. Inone embodiment, the radiographic markers or tie points are stainlesssteel spheres that are {fraction (3/32)}″ in diameter, grade 100material with sphericity within 0.0001 inch and with a hardness ratingof Rockwell C 39. Only three tie points are sufficient to determine acommon plane for merging two 3D maps if they are located perfectly.However some redundancy is desirable and so the present system employs aminimum of four. Although tie points are shown in FIGS. 8A-8B, six tiepoints used in the event that the patient has fillings that way obscuresome of the tie points in the X-ray images.

One procedure for creating the appliance 300 with the radiographic tiepoints 302-308 is discussed next. First, a dental professional takes oneor more PVS impressions and a wax bite for measuring inter-archrelationship. The impressions are used to generate one or more models,which are then scanned. The dental professional then sets up a biteregistration of both arches, and tooth-attachments are mounted onto thedesired teeth which is exactly same size and shape of the actual tiepoints 302-308. In one embodiment, the attachments are mounted on threeteeth in each quadrant (canines, first molars and second molars for atotal of twelve attachments in both arches. Aligners are then fabricatedwith sufficient spaces for retaining the tie points 302-308. The tiepoints 302-308 are positioned onto the spaces, and a thin film ofunfilled composite adhesive is applied to secure the tie points 302-308.Next, the dental professional opens the occlusal surface of aligners toestablish contact between the opposing maxilla and mandible casts.Further, the dental professional can polish the appliance 300 to make itmore comfortable to wear.

FIG. 9 illustrates an exemplary mounting of the tie points on a face350. Points 351 and 352 are located on the forehead above the eyebrowsand are called supercilliary points. Point 353 and 354 are the zygomaticpoints and are located on the anterior aspect of the cheekbone. Points355 and 356 are the gonial tie points and are located at the angle ofthe lower jaw. Point 357 is the nasal tie point and is located on thetip of the nose. Points 359 and 360 are the anterior jaw tie points onebeing located on each side of the chin. Points 362 and 364 are the eartie points located just above the earlobe. In general, at least 3 tiepoints must be visible on at least two images of any set of averages tobe used for merging map components. Further, the tie points shouldsurround the biological region of interest so that measurement to thebiological point can be interpolated. Additionally, the geometry is moreefficient if no more than 2 points are situated along a common line.

In order to avoid the possibility that images of the tie points on theright and the left sides of the face are confused when locating them onlateral X-ray images, the facial tie points on the side of the facenearer to the X-ray emitters 310 and 312 are slightly larger and arepositioned slightly lower on the face than are the facial tie points onthe side of the face nearer to the cassette carrier 318. Similarly, toavoid errors, the tie points on the upper teeth are a slightly differentsize from the tie points on the lower teeth. The information in the tiepoints alone is itself sufficient for the construction of an integratedthree-dimensional craniofacial image map without any specialistintervention or time commitment.

In another implementation, the procedures for the 3D mapping operationare as follows:

(1) Conventional upper and lower impressions are taken and conventionaldental casts are poured and trimmed.

(2) Without damaging the casts, thin cold cure acrylic upper and lowerappliances are fabricated in such a way as to be stably positionable inthe mouth without occlusal interference.

(3) Three or more radiopaque tie points are imbedded into each acrylicappliance with the top points visible at the surface. These are the castcontrol tie points. (Note: steps 4-7 are performed during the samevisit.)

(4) The acrylic appliances are placed in the subject's mouth.

(5) Three or more radiopaque tie points are fixed to the lateral andfrontal aspects of the patient's face. These are the facial control tiepoints.

(6) The patient is positioned in the object space of the stereo X-raysystem as described above and the lateral and frontal X-ray stereopairsare taken as described.

(7) The patient is positioned in the object space of the facialphotographic system and lateral and frontal photographic stereopairs aretaken.

(8) The upper and lower acrylic appliances are returned to theirrespective casts and the casts are individually stereophotographed inappropriate positions.

(9) (Optional) If checks on bite relationship are desired, theappliances may be returned to the mouth and a wax check biteregistration may be made. The appliances and the wax bite are removedfrom the mouth, fastened together and stereo X-rayed. The relationshipof the tie points on this X-ray pair constitutes a mathematicallyapplicable control on the correctness of the bite relationship duringthe taking of the lateral and frontal stereo X-ray pairs.

(10) Appropriate tracings are made of the X-ray and photographicstereopair. Note that each tie point must be located both on thephotographs and on the X-ray images.

(11) The tracing data is reduced to coordinate form.

(12) The two dimensional coordinates of all points used to produce asingle integrated dimensional craniofacial map.

(13) Stored maps from different time points or different subjects may becompared as desired.

Additional supplementations of the stereo X-ray system have been used inprosthetic dentistry (R. Bellagamba, F R Brigante, S. Baumrind (1986) J.Prosthetic Dent. 55:625-28) and in orthopedic surgery for the detectionof pseudarthrosis following lumbar fusions (N. Chafetz, S. Baumrind, J EMorris, H K Genant, E L Korn (1985) Spine 10-368-75) and for thedetection of loosening of femoral prostheses (N. Chafetz, S. Baumrind,W. Murray, H K Genant, E L Kom (1985) Clinical Orthopedics and RelatedResearch 210-60-67).

FIG. 10 shows a data storage and management system for processingintegrated three-dimensional craniofacial analysis from a combination ofdata sources. The system integrates 3D data from stereo cranial X-rays,stereo digital facial images, and digital models of the dentition. Thesystem includes a raw data acquisition module 102. This system acquirespatient data including data from headfilm scanning, stereo images,patient data, digital facial images, study cast images, X-ray images andvideo, collateral material associated with the patient, and treatmentrecords. The output of the raw data acquisition module 102 is providedto a database 104.

The database 104 maintains 2D and 3D coordinates of patient data,computed measures derived from the patient data, digital facial images,study cast images, X-ray images and video, collateral materialassociated with the patient, and treatment records. The database 104also receives data from a film digitizing module 112, which extracts 2Dand 3D coordinate information from the raw images of the patient. Thedatabase 104 can also output data using a publication module 114. Thepublication module 114 can publish image as well as text data for avariety of applications, including teaching and data sharing.

The database 104 can be accessed using a variety of browsers 106, 108and 110. The browser client 106 can view images and data. The browserclient 106 can overlay landmarks upon request, and perform variousanalytical operations. The analysis client 108 performs spatial andtemporal analysis and other statistical operations, among others. Theclient 108 contains one or more analytical software tools for analyzingdata in the database 104. The training client 110 performs traceranalysis and displays ellipses, among others. The client 110 is atraining module for training and calibrating human judges so that theyare consistent in identifying points on images. The display ellipsesstore information relating to acceptable deviations between two judges.As such, the display ellipses improve the quality of the landmarklocations on X-ray images and photographs. Thus, the database 104supports training (calibrating) clinicians and the treatingprofessionals in identifying three-dimensional landmark locations, andintegrating information from pairs of X-ray images to yieldthree-dimensional data and comparing information from multiple timepoints.

FIG. 11 is a simplified block diagram of a data processing system 600that may be used to generate the appliance 300 of FIGS. 3A-3B. The dataprocessing system 600 typically includes at least one processor 602 thatcommunicates with a number of peripheral devices via bus subsystem 604.These peripheral devices typically include a storage subsystem 606(memory subsystem 608 and file storage subsystem 614), a set of user,interface input and output devices 618, and an interface to outsidenetworks 616, including the public switched telephone network. Thisinterface is shown schematically as “Modems and Network Interface” block616, and is coupled to corresponding interface devices in other dataprocessing systems via communication network interface 624. Dataprocessing system 600 could be a terminal or a low-end personal computeror a high-end personal computer, workstation or mainframe.

The user interface input devices typically include a keyboard and mayfurther include a pointing device and a scanner. The pointing device maybe an indirect pointing device such as a mouse, trackball, touchpad, orgraphics tablet, or a direct pointing device such as a touchscreenincorporated into the display, or a three dimensional pointing device,such as the gyroscopic pointing device described in U.S. Pat. No.5,440,326, other types of user interface input devices, such as voicerecognition systems, can also be used. User interface output devicestypically include a printer and a display subsystem, which includes adisplay controller and a display device coupled to the controller. Thedisplay device may be a cathode ray tube (CRT), a flat-panel device suchas a liquid crystal display (LCD), or a projection device. The displaysubsystem may also provide non-visual display such as audio output.

Storage subsystem 606 maintains the basic required programming and dataconstructs. The program modules discussed above are typically stored instorage subsystem 606. Storage subsystem 606 typically comprises memorysubsystem 308 and file storage subsystem 614.

Memory subsystem 608 typically includes a number of memories including amain random access memory (RAM) 610 for storage of instructions and dataduring program execution and a read only memory (ROM) 612 in which fixedinstructions are stored. In the case of Macintosh-compatible personalcomputers the ROM would include portions of the operating system; in thecase of IBM-compatible personal computers, this would include the BIOS(basic input/output system).

File storage subsystem 614 provides persistent (non-volatile) storagefor program and data files, and typically includes at least one harddisk drive and at least one floppy disk drive (with associated removablemedia). There may also be other devices such as a CD-ROM drive andoptical drives (all with their associated removable media).Additionally, the system may include drives of the type with removablemedia cartridges. The removable media cartridges may, for example behard disk cartridges, such as those marketed by Syquest and others, andflexible disk cartridges, such as those marketed by Iomega. One or moreof the drives may be located at a remote location, such as in a serveron a local area network or at a site on the Internet's World Wide Web.

In this context, the term “bus subsystem” is used generically so as toinclude any mechanism for letting the various components and subsystemscommunicate with each other as intended. With the exception of the inputdevices and the display, the other components need not be at the samephysical location. Thus, for example, portions of the file storagesystem could be connected via various local-area or wide-area networkmedia, including telephone lines. Similarly, the input devices anddisplay need not be at the same location as the processor, although itis anticipated that personal computers and workstations typically willbe used.

Bus subsystem 604 is shown schematically as a single bus, but a typicalsystem has a number of buses such as a local bus and one or moreexpansion buses (e.g., ADB, SCSI, ISA, EISA, MCA, NuBus, or PCI), aswell as serial and parallel ports. Network connections are usuallyestablished through a device such as a network adapter on one of theseexpansion buses or a modem on a serial port. The client computer may bea desktop system or a portable system.

Scanner 620 is responsible for scanning casts of the patient's teethobtained either from the patient or from an orthodontist and providingthe scanned digital data set information to data processing system 600for further processing. In a distributed environment, scanner 620 may belocated at a remote location and communicate scanned digital data setinformation to data processing system 600 via network interface 624.Fabrication machine 622 fabricates dental appliances based onintermediate and final data set information received from dataprocessing system 600. In a distributed environment, fabrication machine622 may be located at a remote location and receive data set informationfrom data processing system 600 via network interface 624.

The invention has been described in terms of particular embodiments.Other embodiments are within the scope of the following claims. Forexample, the three-dimensional scanning techniques described above maybe used to analyze material characteristics, such as shrinkage andexpansion, of the materials that form the tooth castings and thealigners. Also, the 3D tooth models and the graphical interfacedescribed above may be used to assist clinicians that treat patientswith conventional braces or other conventional orthodontic appliances,in which case the constraints applied to tooth movement would bemodified accordingly.

What is claimed is:
 1. A method for integrating anatomical informationwith coordinated facial movement and jaw movement from a plurality ofsources of information, comprising: receiving two or more threedimensional (3D) anatomical maps sharing a common plane specified bythree or more marker points common to the two or more maps including oneor more markers embedded in a dental appliance to be worn by a user; andaligning the two or more 3D anatomical maps using the marker points. 2.The method of claim 1, wherein the anatomical information is stereocraniofacial data.
 3. The method of claim 1, wherein one of theanatomical map is an X-ray map.
 4. The method of claim 3, wherein theX-ray map is generated using correlated points on X-ray pairs and usingy-parallax measurements.
 5. The method of claim 3, wherein the X-rayinformation is stereo.
 6. The method of claim 3, further comprisingcalibrating one or more X-ray sources.
 7. The method of claim 6, furthercomprising determining a principal distance from an X-ray source to afilm plane.
 8. The method of claim 6, further comprising characterizinginternal dimensions of the one or more X-ray sources by locating anX-ray film relative to an X-ray source.
 9. The method of claim 1,wherein one of the anatomical map is a 3D image map.
 10. The method ofclaim 1, wherein one of the anatomical map is a dental map.
 11. Themethod of claim 1, wherein each marker is a tie point.
 12. The method ofclaim 1, wherein the aligning uses discrete anatomical landmarkinformation.
 13. The method of claim 1, further comprising displayingthe aligned maps as an integrated 3D anatomical model.
 14. A method forvisualizing anatomical information with coordinated facial movement andjaw movement from a plurality of sources, comprising: receiving X-rayinformation having X-ray marker information including one or moremarkers embedded in a dental appliance to be worn by a user; receiving athree-dimensional teeth model having teeth marker information; aligningthe X-ray information 3d anatomical information, and the 3D teeth modelusing the marker information; and displaying the aligned X-rayinformation, 3D anatomical information, and the 3D teeth model.
 15. Asystem comprising: an X-ray camera receiving X-ray having X-ray markerinformation including one or more markers embedded in a dental applianceto be worn by a user; a three-dimensional digital camera receivingthree-dimensional anatomical information with anatomical markerinformation; a dental scanner to generate a three-dimensional teethmodel with teeth marker information; and a computer to align the X-rayinformation, 3D anatomical information, and the 3D teeth model using themarker information with coordinated facial movement and jaw movement.16. The system of claim 15, wherein the X-ray information is stereo. 17.The system of claim 15, further comprising a calibration array tocalibrate the X-ray camera.
 18. The system of claim 15, furthercomprising an X-ray cassette carrier.
 19. The system of claim 15,wherein each marker is a tie point.
 20. The system of claim 15, whereinthe computer uses discrete anatomical landmark information.